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1.  ABSTRACT 

We  are  developing  a  mobile  handheld  system  that 
provides  tactical  and  cultural  advice  to  warfighters.  The 
system  fuses  built-in  knowledge  of  social  factors  and 
location  specific  information  with  dynamically  entered 
situation  descriptions  to  produce  an  assessment  of  the 
situation  and  recommend  actions.  The  outcomes  include 
the  system’s  confidence  in  its  recommendations,  along 
with  explanations  of  its  reasoning.  The  result  gives 
warfighters  the  information  they  need  to  make  the  critical 
decisions  necessary  to  accomplish  their  mission,  while 
minimizing  the  collateral  damage  that  alienates  the  local 
populace. 

2.  INTRODUCTION 

According  to  the  Defense  Advanced  Research 
Projects  Agency,  the  key  to  success  in  the  urban 
battlespace  is  the  development  of  an  understanding  of  the 
relationships  among  the  civilian  population,  military 
forces,  and  urban  infrastructure  that  define  the  battle 
context.  There  are  various  systems  in  development  that 
will  effectively  expose  urban  infrastructure  to  enable 
effective  military  tactical  development.  What  these 
systems  lack  in  establishing  battlespace  success  is  cultural 
awareness.  The  goal  of  the  ACTAR  system  (Augmented 
Cultural  Tactical  Awareness  and  Response)  is  to  establish 
the  desired  interrelationship  between  these  progressive 
systems  and  the  populace  of  the  battlespace  by  enabling 
cultural  tactical  analysis. 

The  necessary  management  of  an  urban  battlespace 
will  depend  on  a  soldier’s  ability  to  amalgamate  and 
empathize  with  the  resident  culture.  This  requires 
awareness  of  the  culture  in  question.  Augmenting  cultural 
awareness  entails  integrating  a  soldier’s  current  tactical 
perspective  with  a  resource  for  assessing  various  cultural 
aspects  relevant  to  the  scenario,  such  as 

•  Pentagon  policies, 

•  local  political  exigencies, 

•  cultural  norms,  traditions,  and  expectations,  and 


•  sociological,  religious,  legal,  geographic, 
economical,  and  historical  factors. 

The  urban  model  generated  by  this  culturally 
enhanced  perspective  will  incorporate  an  understanding  of 
the  area’s  populace,  resulting  in  sensitivity  to  the 
inhabitants  of  the  urban  battlespace  that  is  currently  not 
considered  by  existing  systems  for  devising  tactical 
operations.  Establishing  sensitivity  to  the  populace  of  an 
urban  battlespace  via  the  ACTAR  system  will  enable  a 
more  predictable  projection  of  urban  activity. 

There  are  two  primary  ways  in  which  ACTAR  will 
facilitate  a  predictable  projection  of  urban  activity. 

1 .  Consideration  for  the  concerns  of  the  populace 
will  enhance  the  relationship  between  soldiers 
and  civilians.  A  good  relationship  will  facilitate 
more  predictable  and  constructive  behavior  by 
both  parties. 

2.  Taking  into  account  the  normal,  traditional,  or 
expected  behavior  of  a  culture  of  people  will 
enable  a  characterization  of  suspicious  behavior 
with  respect  to  the  culture  of  interest. 

As  such,  the  key  to  success  in  the  urban  battlespace  is 
knowledge — knowledge  of  the  techniques  and  limitations 
of  the  opponents;  knowledge  of  the  current  location  of 
enemies,  friends,  and  weapons;  knowledge  of  the 
interrelationships  among  the  opponents;  and  knowledge 
of  the  urban  terrain.  Before  a  soldier  enters  a  building,  he 
should  know  exactly  whether  he  should  expect  to  find 
opposing  forces,  civilians,  or  both,  how  to  communicate 
with  them,  what  their  reaction  to  his  presence  will  be,  and 
what  type  of  response  to  expect. 

We  believe  that  most  of  the  technology  for  delivering 
and  acquiring  this  information  already  exists,  but  what  is 
needed  is  the  distributed  software  that  will  bring  together 
all  the  disparate  sensing  and  communication  technologies. 
We  are  thus  developing  software  for  handheld  devices, 
which  will  record  and  aggregate  the  observations  of  the 
warfighters  in  the  field  and  provide  them  with  specific, 
time-  and  location-sensitive  advice.  Our  handheld 
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application  will  tell  a  soldier  what  he  needs  to  know  when 
he  needs  to  know  it,  all  the  while  minimizing  the  amount 
of  interaction  the  soldier  needs  to  have  with  our  system  so 
that  he  can  be  free  to  carry  out  his  mission.  The  ultimate 
goal  of  our  system  is  to  give  each  soldier  omniscience 
over  the  urban  battlespace.  This  will  be  accomplished  by 
a  priori  aggregation  of  cultural  and  command-and-control 
knowledge  and  in  situ  collaborative  aggregation  of  state 
knowledge  by  the  warfighters,  sensors,  and  any  other 
autonomous  systems  (e.g.,  UAVs  and  robots)  available. 

We  are  developing  a  mobile  handheld  system  that 
provides  tactical  and  cultural  advice  to  warfighters.  Our 
project  consists  of  two  major  parts:  (1)  we  are 
constructing  a  cultural  and  military  adviser  for 
warfighters  that  operates  independently,  without  the  need 
for  communications  with  other  handhelds,  to  provide 
warfighters  with  specific  advice  on  what  to  expect  given 
their  current  situation,  the  cultural  imperatives  of  their 
opponents,  and  orders  from  command;  and  (2)  we  are 
utilizing  the  wireless  capabilities  of  the  handheld  in  order 
to  let  warfighters  cooperate  in  creating  an  aggregate  view 
of  the  urban  battlespace.  The  system  fuses  built-in 
knowledge  of  social  factors  and  location-specific 
information  with  dynamically  entered  situation 
descriptions  to  produce  an  assessment  of  the  situation  and 
recommend  actions.  The  outcomes  include  the  system’s 
confidence  in  its  recommendations,  along  with 
explanations  of  its  reasoning.  The  result  gives  warfighters 
the  information  they  need  to  make  the  critical  decisions 
necessary  to  accomplish  their  mission,  while  minimizing 
the  collateral  damage  that  alienates  the  local  populace. 

3.  A  NEW  APPROACH  IN  THE  URBAN 
BATTLESPACE 

Imagine  a  group  of  suspected  terrorists  has  escaped 
into  a  religious  building,  such  as  a  mosque,  temple, 
church,  or  synagogue.  Should  they  be  pursued  into  such  a 
building?  If  so,  what  are  the  social  or  political 
ramifications  that  might  negatively  affect  the  larger 
mission?  Are  there  any  other  effective  options  available  to 
capture  the  terrorists  that  may  mitigate  the  negative 
consequences?  What  if  the  building  in  question  is  a 
different  kind  of  public  place,  such  as  a  school, 
marketplace,  hospital,  government  office,  or  historic  site? 
Do  the  implications  of  actions  change  if  the  context  is 
more  personal,  such  as  a  private  home  or  a  graveyard  or 
mausoleum?  Does  the  day  of  the  week,  such  as  a  holy  day 
when  religious  services  are  being  held,  affect  the 
decision?  Is  the  gender  of  the  suspects  relevant?  Does 
their  age  matter? 

These  are  complex  questions  requiring  diverse  and 
expert  knowledge  to  support  complex  decision-making.  A 
warfighter  in  an  urban  battlespace  must  consider  all  of 


these  factors  while  making  tactical  decisions,  and  often 
under  extreme  duress  and  danger.  To  add  to  the  difficulty, 
most  of  the  factors  are  more  qualitative  than  quantitative, 
and  thus  are  difficult  to  assess,  combine,  and  compare. 
There  is  no  time  to  assemble  and  consult  a  team  of 
experts,  yet  the  decision  must  none-the-less  be  based  on 
accurate  and  appropriate  data  that  informs  and  minimizes 
the  problem  space. 

The  warfighters  obviously  need  help  with  making 
decisions  that  take  all  of  these  factors  into  account,  but  no 
computational  aids  are  available.  Warfighters  have  C4I 
support  systems  that  direct  them  in  battlefield  tactics,  yet 
they  are  completely  unequipped  in  effectively  handling 
scenarios  such  as  those  described  above.  The  available 
support  systems  are  oriented  around  winning  the  battle. 
Effectively  handling  scenarios  like  those  described  above 
will  lead  to  the  mitigation  of  tensions  and  the  fostering  of 
relationships.  These  are  acts  of  peace.  What  warfighters 
now  require  is  a  tool  for  winning  peace. 

4.  THE  NEED  FOR  CULTURAL  AWARENESS: 
BRIDGING  THE  GAP 

Culture  is  an  evolved  capacity  of  the  human  species. 
As  with  any  evolution,  there  is  a  specific  environmentally 
motivated  justification  for  why  we  have  evolved  such  a 
complicated  characteristic  such  as  culture.  What  does 
culture  provide  to  human  beings,  and  why  is  it 
increasingly  important  in  today’s  world  to  facilitate  the 
generation  of  cross-cultural  relationships?  In  this  section 
we  will  briefly  define  culture,  and  describe  how  it  helps 
and  hinders  our  ability  to  get  along.  We  will  also  discuss 
how  a  computational  system  enabling  cross-cultural 
integration  can  assist  in  significantly  mitigating 
intercultural  conflict. 

Communities  of  people  are  able  to  establish  common 
behaviors  that  are  explicitly  taught  and  indirectly  shared 
with  others  within  the  community  (Linton,  1945). 
Lederach  (1995)  agrees  with  Linton  that  culture  is  “shared 
knowledge  and  schemes  created  by  a  set  of  people,”  then 
he  continues  by  specifying  reason  for  creating  such 
commonalities,  namely  “for  perceiving,  interpreting, 
expressing,  and  responding  to  the  social  realities  around 
them.”  Thus,  culture  is  a  human  response  in  which  people 
establish  common  behaviors  that  are  expected  in  certain 
social  contexts.  Kroeber  and  Kluckhohn  (1952)  declare 
that,  “culture  systems  may,  on  the  one  hand,  be 
considered  as  products  of  action,  and  on  the  other  as 
conditioning  elements  of  further  action.”  If  this  cultural 
behavior  is  presented  in  appropriate  contexts,  it  creates 
congruence  between  the  actors  generating  predictable 
subsequent  behaviors,  which  may  be  useful  in 
establishing  pro-social  relationships  such  as  trust  or 
comfort.  Imagine  the  following  typical  greeting  in  US 
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culture:  if  someone  approaches  you  with  an  outstretched 
hand  and  a  smile  on  their  face,  there  is  common  protocol 
as  to  the  behavior  that  will  ensue.  The  common  protocol 
puts  the  one  being  greeted  at  ease,  and  this  ease  diffuses 
any  tensions  that  may  have  existed.  On  the  contrary, 
imagine  if  the  greeter  bluntly  got  to  the  point  saying 
abrasively  “Who  are  you?”  Tensions  would  be  raised  as  a 
result  of  this  broken  cultural  protocol. 

Just  as  culture  generates  pro-social  opportunities, 
lack  of  cultural  awareness  may  have  anti-social 
consequences.  Lederach  (1995)  describes  the  relationship 
between  culture  and  conflict  as  one  in  which  “social 
conflict  emerges  and  develops  on  the  basis  of  the  meaning 
and  interpretation  people  involved  attach  to  action  and 
events...  From  this  starting  point,  conflict  is  connected  to 
meaning,  meaning  to  knowledge,  and  knowledge  is  rooted 
in  culture.”  Culture  is  in  essence  at  the  root  of  conflict, 
generating  conflict  because  culture  is  missing  and  so 
meaning  is  missing.  Two  groups  of  people  that  have  no 
shared  knowledge  or  mutual  understanding  will  inevitably 
conflict  as  one  group  remains  unaware  of  the  social 
realities  to  which  the  other  is  sensitive. 


the  assistant  could  provide  contextually  and  strategically 
appropriate  suggestions  to  a  warfighter. 
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Fig.  1  ACTAR  system’s  components 


Cross-cultural  integration  is  essential  for  bridging  the 
cultural  divide  and  establishing  genuine  and  effective 
understanding  between  cultures.  Once  cultural  awareness 
is  generated,  unnecessary  and  unintentional  conflicts 
resulting  from  ontological  incongruence  will  be  mitigated. 
Computer  systems  can  be  a  useful  means  of  establishing 
this  understanding  in  a  timely  and  accurate  manner.  The 
system  will  not  simply  convey  cultural  data;  it  will 
identify  appropriate  cultural  actions  the  user  should 
consider  taking  with  detailed  explanations  delineating 
why  such  an  action  would  be  appropriate. 

A  computer  system  may  be  more  aware  of  the  details 
of  the  battlespace  than  the  soldier.  This  is  no  surprise,  as 
the  system  is  able  to  store  more  information  than  the  user, 
and  hence  allow  the  user  the  flexibility  of  focusing  on  a 
more  immediate  sub-context.  The  important  factor  to  this 
is  that  the  system  must  be  aware  of  when  certain 
information  is  pertinent  to  the  user’s  localized  battlespace. 
In  this  way,  a  computer  system  may  suggest  culturally 
motivated  actions  to  the  user  that  are  based  on  apparently 
extraneous  data  such  as  where  other  troop  members  are 
located  in  the  battlespace  or  the  current  suspect’s  recorded 
history.  People  have  only  a  limited  capacity  to  consider 
distant  and  seemingly  unrelated  consequences  to  an 
action,  particularly  when  these  actions  have  to  do  with  a 
foreign  culture  of  which  the  person  has  minimal 
understanding  (Domer,  1996).  Computers  can  provide 
significant  assistance  in  this  regard.  A  culturally 
motivated  decision-making  computational  assistant  such 
as  ACTAR  can  manage  the  cultural  information  as  well  as 
an  awareness  of  the  battlespace.  In  processing  this  data, 


The  ACTAR  system  is  designed  for  Windows  Mobile  5.0 
powered  handheld  devices  utilizing  the  .NET  Compact 
Framework.  Its  architecture  is  conceptually  composed  of 
several  subsystems  or  modules  that  provide  services  to 
each  other.  Figure  1  shows  the  components  of  the  system 
and  the  communication  links  among  them.  Each 
subsystem  is  in  charge  of  a  particular  task,  as  described 
next. 

♦>  The  GPS  Manager  (GPSM)  controls  the  handheld 
GPS  hardware.  It  is  responsible  for  retrieving  GPS 
coordinates  and  providing  GPS  power  management 
services. 

♦>  The  Map  Engine  (ME)  provides  the  mapping 
services  for  the  ACTAR  system.  It  is  responsible  for 
rendering  GPS  maps  and  keeping  track  of  map 
landmarks,  such  as  buildings  and  points  of  interest.  It 
requests  the  services  of  the  GPSM  to  calculate  GPS 
coordinates. 

❖  The  Communications  module  (COMMS)  manages 
handheld-to-handheld  communication,  interactions 
with  a  central  command  center,  and  control  and  data 
exchanges  with  sensors.  The  channels  are  provided 
by  WiFi,  BlueTooth,  and  USB. 

♦>  The  Cultural  Knowledge  Base  (CKB)  contains 
Bayesian  models  of  command  and  cultural  domain 
knowledge  encoded  as  cases.  The  models  correspond 
to  situations  where  cultural  information  is  a  key  to 
success.  Each  case  contains  a  set  of  attributes  that 
describe  the  situation  and  a  recommendation  for  that 
situation.  Examples  of  these  attributes  are: 
Information  about  the  scenario  (building  search, 
personal  search,  etc.),  who  is  involved  (males, 
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females,  infants,  children,  etc.),  where  is  the  situation 
occurring  or  located  (buildings,  street,  residence, 
etc.),  when  (prayer  time,  dinner  time,  day,  night, 
etc.),  and  the  perceived  threat  (suspected  bomb 
making  location,  suspected  terrorist/insurgent, 
suicide  bomber,  civil  unrest,  death  of  insurgent,  etc.). 
Each  attribute  has  a  weight  or  probability  associated 
with  it.  The  CKB  module  can  be  changed  according 
to  the  culture  where  the  warfighter  is  carrying  out  a 
mission.  For  example,  we  can  develop  a  CKB 
module  for  different  cultures1  (e.g.,  Middle  East, 
Eastern  Europe,  Central  Africa,  and  South  America) 
and  load  the  corresponding  CKB  in  the  ACTAR 
system  before  leaving  for  a  mission,  depending  in  the 
culture  of  the  mission’s  location. 

♦>  The  Data  Manager  (DM)  aggregates  sensed  data, 
data  from  the  warfighter,  geographical  information 
given  by  the  ME  and  the  GPSM,  information  about 
the  military  unit  of  the  warfighter  (including  the 
number  of  its  members,  armaments  and  devices 
available,  etc.),  and  mission  details  (target,  goal, 
location,  etc.). 


some  of  the  attributes  of  the  case  are  missing,  it  asks 
the  warfighter  about  the  attributes  using  an  intuitive 
data  entry  form  in  the  UI.  The  answers  improve  the 
accuracy  of  the  recommendation.  The  ultimate  choice 
of  decision  is  the  one  that  maximizes  the  expected 
benefit  of  a  decision  (utility  -  cost),  as  given  by 
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❖  The  User  Interface  (UI)  enables  control  and 
communication  between  the  user  and  the  other 
components  of  the  system. 


Figure  3  shows  part  of  the  UI  of  the  ACTAR  system. 
The  yellow  icon  in  the  map  represents  the  position  of  the 
warfighter  or  user  of  the  system  and  the  red  icon 
represents  a  target  the  user  has  indicated  on  the  map. 
Figure  4  shows  an  example  of  ACTAR’ s 
recommendations.  In  this  case,  the  situation  is  a  personal 
search  of  an  adult  male  suspected  of  being  a  suicide 
bomber,  while  the  suspect  is  in  a  mosque  at  prayer  time. 


Fig.  2  The  decision  mechanism  in  ACTAR 


❖  The  Reasoning  Engine  (RE)  provides 
recommendations  to  improve  the  decisions  of  the 
warfighters  according  to  the  cultural  context  of  the 
situation.  It  executes  case-based  reasoning  over 
influence  diagrams,  as  shown  in  Figure  2,  using  the 
models  in  the  CKB  with  the  user’s  information  and 
the  aggregated  data  provided  by  the  DM.  When  the 
RE  finds  the  closest  case  to  the  current  situation  and 


Fig.  3  ACTAR  system’s  user  interface. 


1  Currently  we  have  implemented  only  one  CKB  module 
for  the  Arab  culture. 
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6.  RESEARCH  CHALLENGES 


A  cultural  decision-making  advisory  system  for 
military  domains  poses  several  significant  research 
challenges  on  the  path  to  attaining  this  vision.  In  this 
section  we  introduce  and  briefly  describe  each  one. 

Our  design  involves  the  integration  and 
accommodation  of  various  disparate  representations  of 
social  information,  reasoning  methods,  and  abstractions 
for  individual  contextual  factors.  For  example,  if  a  soldier 
on  patrol  at  night  encounters  a  mother  with  a  sick  child 
breaking  curfew,  the  advisor  will  begin  integrating  the 
contextual  factors  of  the  broken  curfew,  the  sick  child, 
and  the  known  hours  of  operation  of  local  hospitals  with 
reasoning  methods  that  will  result  in  the  assessment  that 
the  child  needs  to  go  to  the  nearest  hospital.  This 
information  will  instantly  be  reasoned  with  physical 
information,  such  as  GPS  coordinates  of  the  nearest 
hospital  in  the  area,  as  well  as  social  information  as  to 
how  to  interact  with  the  likely  terrified  mother  as  to 
assuage  her  fears  and  establish  comfort  or  possibly  trust 
in  the  soldier  who  is  there  to  help  her  child.  These  various 
reasoning  methods  must  work  together  to  create  a  holistic 
understanding  of  the  domain  space  based  on  the  social, 
physical,  and  other  contextual  information  being  gathered. 
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Explanation 


Seek  permission  to  enter  Mosque. 

In  some  areas,  non-Muslims  are  not 
allowed  into  Mosques,  but  in  others 
non-Muslims  may  only  enter  mosques  if 
granted  permission  to  do  so  by  Muslims 
and  if  they  have  a  legitimate  reason. 
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Fig.  4  Example  of  recommendations  provided  by 
ACTAR. 


Acquiring  domain  knowledge  for  each  factor  will  be 
challenging.  Some  information  poses  little  resistance, 
such  as  using  Web  services  to  extract  Web-accessible 
data,  such  as  GPS  coordinate  analysis  or  weather 
forecasts  and  traffic  reports  in  certain  areas.  Other 
information  will  be  significantly  more  difficult  to  gather 
and  maintain.  For  example,  social  networks  of  people  are 
often  oral  creations  that  are  generated  and  maintained 
through  common  events  and  acquaintances.  This  kind  of 
information  is  often  difficult  to  digitize.  For  the  particular 
example  of  social  networks,  notably,  much  work  has  been 
done  in  the  domain  of  counterterrorism  with  respect  to 
identifying  and  modeling  dynamic  social  networks 
(Behrman  and  Kathleen,  2003). 

The  difficulty  of  extracting  domain  knowledge  is 
coupled  with  the  issue  of  having  to  represent  this  data 
within  the  appropriate  time  constraints.  Outdated  data  is 
more  than  useless:  it  is  wasteful.  As  complicated  and 
resource  intensive  as  it  is  to  define  a  dynamic  social 
network,  if  this  network  does  not  provide  useful 
information,  i.e.,  the  right  data  at  the  right  time,  then  the 
expense  of  calculation  is  to  no  avail.  Specific  situations 
will  have  to  be  represented  fast  enough  and  clearly 
enough  for  contextual  decisions  to  be  made  and  actions  to 
be  taken  in  a  timely  manner  to  ensure  the  validity  of  the 
data  being  acted  upon.  Generating  this  caliber  of 
representation  and  managing  its  reliability  is  a  challenge 
of  this  work. 

Once  the  social  information  has  been  gathered  and 
processed,  the  decision  support  system  must  then  present 
its  recommended  courses  of  action  to  the  warfighter,  who 
will  then  use  it  to  make  decisions.  This  poses  a  research 
issue  in  the  area  of  human-computer  interfacing.  The 
system  will  want  to  present  the  information  concisely  yet 
clearly,  so  the  warfighter  does  not  simply  see  a  list  of 
possible  actions,  but  a  description  of  the  actions,  their 
projected  outcomes  and  explanations  of  the  analysis,  as 
well  as  comparisons  among  possible  actions.  In  various 
scenarios,  visual  representations  of  the  analysis  will  also 
be  an  effective  means  of  relating  information  to  the  user. 


7.  DEPLOYMENT: 

IMPACT  FOR  THE  WARFIGHTER 

The  mobile  handheld  cultural  advisor  will  provide 
several  new  critical  benefits  to  a  warfighter.  First,  it  will 
enable  the  integration  of  human-sensed  information  into 
consistent  and  concise  aggregate  views  for  commanders, 
as  indicated  in  Figure  5.  Second,  it  will  provide  situation- 
specific  advice  to  a  warfighter  by  leveraging  the  various 
sensors  that  are  connected  to  the  handheld,  the  soldier’s 
own  inputs  to  the  handheld,  and  the  Bayesian-encoded 
models  we  create  of  command  and  domain  knowledge. 
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We  envision  having  various  knowledge  modules,  such  as 
an  Arab  module,  a  Latin  American  module,  and  a  Central 
African  module.  Each  would  incorporate  different  domain 
knowledge  from  experts  in  that  particular  culture.  The 
results  of  our  research  are  contributing  to  an  improved 
understanding  of  representation  and  reasoning  in  complex 
social  situations  and  will  improve  the  decisions  of 
warfighters  by  guiding  their  choice  of  the  appropriate 
tactics  in  the  appropriate  context. 

7.1  The  Advisor 

One  of  the  major  obstacles  that  we  find  in  the  urban 
battlespace  comes  from  our  warfighters'  lack  of 
knowledge  about  the  specific  cultural  imperatives  and 
guerrilla  techniques  of  their  host  country.  It  is  also  hard 
for  fresh  soldiers  to  remember  every  single  piece  of 
advice  from  their  training  and  apply  it  when  needed.  As 
such,  we  propose  to  develop  software  for  handhelds  that 
can  be  used  by  a  soldier  to  get  useful  context-sensitive 
information  on  what  to  do  or  what  response  to  expect 
given  his  actions.  A  key  part  of  our  proposal  is  that  this 
information  must  be  situation  specific  so  as  not  to 
overload  the  soldier  with  needless  information. 

Of  course,  in  order  to  provide  situation  specific 
information,  our  software  will  need  to  know  the  situation 
in  which  the  soldier  finds  himself.  Another  key  idea  in 
our  design  is  to  maximize  the  use  of  passive  sensing  and  a 
priori  data  entry,  such  that  the  soldier  will  need  to  input  as 
little  information  as  possible  when  he  needs  advice.  For 
example,  before  going  out  on  patrol,  the  soldier  could  tell 
our  software  the  planned  path,  the  number  of  warfighters 
in  the  patrol,  their  abilities,  weapons,  equipment,  etc.  For 
the  passive  sensors,  we  plan  to  use  the  handheld's  clock  to 
determine  the  time  and  its  microphone  to  determine  if  the 
soldier  is  currently  under  fire.  Its  GPS  sensor  will  tell  our 
system  the  soldier's  exact  location.  In  future  releases,  we 
envision  the  addition  of  other  simple  sensors.  For 
example,  the  soldier  could  carry  a  tiny  digital  camera  that 
would  automatically  take  pictures  of  what  he  sees  and 
wirelessly  deliver  these  to  his  handheld.  The  soldier  could 
be  outfitted  with  throwable  disposable  cameras  that  he 
could  place  in  strategic  locations  so  as  to  know  what  is 
happening  there  even  after  he  leaves  or  he  could  throw 
them  over  walls,  slide  them  under  doors,  throw  them 
across  open  windows,  etc.  to  see  parts  of  the  battlespace 
that  would  otherwise  remain  hidden  to  him.  Similarly, 
there  could  be  disposable  motion  sensors  that  the  soldier 
places  in  strategic  locations  to  alert  him  of  nearby 
movement,  thus  acting  as  an  electronic  tripwire.  Once 
GPS  systems  become  more  miniaturized  we'll  have 
disposable  tracking  devices  that  the  soldier  can  place  on 
suspect  vehicles  or  persons  in  order  to  track  their 
movement  across  the  urban  battlespace.  These  are  just 
some  of  the  possible  passive  sensors  that  already  exist, 
even  if  their  current  size  and  pricing  might  be  too  large. 


We  expect  that  even  more  creative  sensors  will  be 
developed  in  the  near  future  using  research  from  micro- 
fabricated  MEMS  and  nanotechnology. 

All  these  sensors,  however,  are  useless  without  a  way 
for  aggregating  automatically  the  data  they  provide  and 
making  it  available  to  the  soldier  using  an  intuitive 
interface.  Our  handheld  adviser  serves  this  role  by 
becoming  the  local  hub  for  all  the  data  provided  by  the 
various  sensors.  It  is  easy  to  physically  deliver  the  raw 
data  to  the  handheld  using  encrypted  Bluetooth  or  WiFi 
channels.  However,  aggregating  data  from  various  sensors 
into  a  coherent  picture  of  the  current  state  of  the  world 
and  then  offering  advice  to  the  soldier  based  on  this  state 
is  a  more  challenging  task.  This  is  the  task  we  propose  to 
accomplish. 

Our  handheld  software  will  examine  the  soldier's 
current  state  along  with  any  specific  query  the  soldier 
might  pose,  and  deliver  situation- specific  advice  based  on 
deep  domain  and  command  knowledge.  At  its  simplest, 
our  software  will  tell  the  soldier,  given  his  current 
situation,  which  actions  will  result  in  surprising 
consequences,  under  the  assumption  that  the  soldier  is  not 
familiar  with  the  social  mores  of  his  host  country.  This 
advice  could  either  pop  up  as  alerts,  if  especially 
important,  or  remain  hidden  until  requested  by  the  solder. 
That  is,  the  soldier  can  query  our  system  on  the  expected 
results  of  a  particular  action.  If  the  system’s  uncertainty  is 
high,  it  might  give  preliminary  advice  and  ask  the  soldier 
for  more  information  in  order  to  increase  the  certainty. 
For  example,  given  the  current  state,  the  system  might 
conclude  that  it  is  somewhat  likely  that  the  seemingly 
non-threatening  people  the  soldier  observes  are  really 
armed  opponents.  The  system  could  then  give  the  soldier 
a  question,  in  the  local  language,  that  he  can  ask  the 
unidentified  individuals.  Their  response  to  the  question 
will  determine,  with  high  certainty,  whether  they  are 
friend  or  foe.  More  generally,  our  system  would  provide 
advice  on  how  to  handle  a  wide  variety  of  common 
situations,  including  confrontations  with  mixed  crowds 
composed  of  women,  children,  and  possible  insurgents. 

When  the  cultural  and  command  knowledge  is 
encoded  into  Bayesian  networks  that  capture  the  set  of 
possible  states  a  soldier  might  be  in,  then  the  networks 
will  give  us  probability  estimates  on  hidden  aspects  of  the 
state,  that  is,  they  will  induce  unobservable  aspects  of  the 
world.  We  will  also  encode  the  expected  outcomes  of 
possible  actions  in  every  possible  state.  Once  we  have  all 
this  information  encoded  it  is  a  simple  matter  to  calculate 
the  expected  outcomes  of  every  action  available  to  the 
soldier  given  the  evidence  and  tell  him  which  actions  are 
particularly  good  or  bad.  In  other  words,  we  are  building  a 
sophisticated  Bayesian  model  of  the  complex 
interrelations  among  the  various  actors  in  an  urban 
battlespace,  taking  into  account  their  social,  economic, 
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political,  and  military  attributes.  This  model  can  then  be 
used  to  determine  the  expected  outcome  of  a  soldier's 
action. 

Calculating  the  best  action,  or  even  just  eliminating 
bad  actions,  also  require  us  to  use  and  expand  techniques 
from  applied  game  theory.  Our  model  is  both  probabilistic 
and  large.  The  soldier's  state  is  rarely  completely  known 
and  the  immediate  result  of  actions,  both  the  soldier's  and 
the  opponents',  can  also  be  probabilistic.  As  such,  our 
system  will  need  to  calculate  equilibria  for  large  non- 
deterministic  extended- form  games.  These  problems  are 
generally  intractable  for  everything  but  the  smallest 
instances.  We  overcome  this  difficulty  by  developing 
algorithms  that  use  our  extensive  domain  knowledge  to 
prune  unlikely  situations  and  actions.  We  also  recognize 
that  while  there  might  be  a  large  number  of  actors,  each 
one  has  social  relations  with  only  a  few  others.  Thus,  we 
do  not  need  to  worry  about  how  one  agent's  actions  might 
affect  everyone.  This  greatly  reduces  the  dimensionality 
of  the  problem — it  creates  sparse  payoff  matrices  and 
thinner  extended-form  trees  that  are  easier  to  solve. 

7.2  Collaborative  Information  Gathering  and  Analysis 

Most  handhelds  on  the  market  today  come  with  WiFi 
and  Bluetooth  built-in.  The  moment  that  handhelds  can 
communicate  with  neighboring  handhelds  we  open  a  very 
large  space  of  possible  collaborative  applications. 
Specifically,  while  it  is  impossible  for  the  lone  soldier  to 
get  a  complete  view  of  the  state  of  the  urban  battlespace 
by  himself,  we  believe  that  an  army  of  warfighters  with 
handhelds  can  collaboratively  achieve  omniscience  over 
the  urban  battlespace.  Each  soldier  can  contribute  what  he 
or  his  sensors  perceive.  Furthermore,  the  system  can 
request  that  the  warfighters  make  observations  that 
benefit  the  group  as  a  whole,  even  if  they  might 
inconvenience  an  individual  soldier.  That  is,  the 
warfighters  are  not  just  passive  sensors,  they  become 
active  information-gathering  agents  at  the  service  of  a 
command.  For  example,  the  system  might  note  that  a  unit 
might  be  in  danger  from  possible  sniper  attack,  determine 
their  possible  locations,  determine  which  other 
warfighters  in  the  area  have  visibility  over  those 
locations,  and  finally  ask  these  warfighters  to  turn  around 
and  look  at  those  locations  to  check  for  suspicious 
movement. 

The  global  picture  of  the  battlespace  that  is  thus 
generated  can  also  be  relayed  to  central  command  where 
experts  can  make  highly  informed  decisions  about  how  to 
proceed.  Their  orders  or  suggestions  are  propagated  back 
to  the  warfighters  in  the  form  of  added  certainty  about 
some  element,  “that  figure  is  definitely  an  insurgent,”  or 
new  knowledge  about  what  action  to  take  in  certain 
situations,  e.g.,  “zone  X  has  been  compromised,  avoid  it 
at  all  costs.” 


The  software  developed  for  this  aspect  of  the  system 
includes  a  map-based  interface,  which  will  be  especially 
useful  as  it  can  naturally  limit  a  soldier’s  view  to  only 
those  things  that  are  nearby.  It  enables  a  warfighter  to 
enter  information  on  a  map  using  gestures.  He  can  drag 
and  drop  icons  that  represent  possible  enemies,  possible 
booby  traps,  possible  allies,  and  any  other  point  of  interest 
or  draw  circles  around  areas  he  considers  dangerous, 
areas  where  there  are  civilians,  areas  of  strategic 
importance,  etc.  Each  data  point  will  be  annotated  with 
the  time  it  was  observed,  the  owner  of  that  observation, 
and  any  other  details  pertinent  to  it. 

The  observations  are  then  automatically  distributed 
among  the  warfighters  by  the  system,  thus  allowing  a 
soldier  to  see  everything  everyone  else  is  seeing. 
Furthermore,  our  system  will  reconcile  conflicting 
evidence  by  using  probability  formulas  and  display  these 
probabilistic  states  in  an  intuitive,  graphical  manner.  For 
example,  if  there  are  conflicting  reports  on  the  possibility 
of  a  sniper  at  the  top  of  a  building  then  the  icon  will 
reflect  this  uncertainty,  perhaps  by  using  different  shades 
of  red.  Our  system  will  apply  its  acquired  knowledge  as 
represented  by  the  icons  to  create  personalized  views.  For 
example,  if  a  soldier  is  walking,  then  the  areas  that  are 
within  gunshot  of  a  reported  sniper  position  will  be 
highlighted  as  dangerous.  Finally,  the  system  will  apply 
its  knowledge  about  the  icons  to  make  useful 
representations  at  various  zoom  levels.  For  example,  what 
appears  as  individual  insurgents  at  a  high  zoom  turns  into 
a  red  zone  as  we  zoom  out,  where  the  red  zone  indicates 
areas  of  unrest. 

In  effect,  we  view  an  army  of  warfighters  with 
handhelds  as  a  distributed  sensor  network  where  each 
soldier  is  an  intelligent  autonomous  sensor.  The  fact  that 
the  warfighters  are  intelligent  means  that  they  can 
interpret  raw  sensor  data,  for  example,  they  can  tell  the 
system  if  a  loud  noise  was  a  gunshot  or  a  car  backfiring. 
The  fact  that  the  warfighters  are  autonomous  means  that 
our  system  will  need  to  take  into  account  the  fact  that  it 
can  only  ask  them  to  take  certain  actions,  but  ultimately 
the  individual  warfighter  must  decide  what  to  do.  Also, 
human  limitations  mean  that  there  will  be  inconsistencies 
in  the  sensed  information.  One  soldier  might  identify  an 
enemy  in  the  second  floor  of  a  building,  while  another 
soldier  might  claim  that  there  is  nobody  there. 

In  the  future,  we  envision  warfighters  using  heads-up 
displays  that  provide  them  with  an  augmented  reality 
view  of  the  urban  battlespace  and  a  voice  recognition 
interface  to  communicate  with  our  system.  Note, 
however,  that  while  these  input/output  technologies  are 
still  not  ready  for  use,  the  software  we  propose  to  develop 
should  not  need  to  change  much  as  we  transition  from 
handhelds  to  heads-up  displays.  The  problem  of  data 
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aggregation  is  complex  and  it  is  one  that  we  can  and 
should  start  working  on  now,  even  if  the  current 
technologies  for  its  use  are  sometimes  cumbersome. 
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Fig.  5.  Schematic  view  of  handheld  advisors’  (ACTAR’s)  deployment 


8.  CONCLUSION 

In  summary,  our  handheld  adviser  can  provide 
situation- specific  advice  to  the  warfighter  by  leveraging 
the  various  sensors  that  are  connected  to  the  handheld,  the 
warfighter's  own  inputs  to  the  handheld,  and  the  Bayesian 
network  models  we  create  of  command  and  domain 
knowledge.  We  envision  having  various  knowledge 
“modules,”  such  as  an  Arab  module,  a  Latin  American 
module,  an  urban  battlespace  module,  and  a  suburban 
battlespace  module.  Each  one  would  incorporate  different 
domain  knowledge  from  experts  in  that  particular  culture 
or  battlespace.  Warfighters  would  then  be  able  to  choose 
to  install  only  the  modules  they  need.  The  results  of  our 
development  are  contributing  to  an  improved 
understanding  of  representation  and  reasoning  in  complex 
social  situations  and  will  have  the  practical  benefits  of 
improving  the  decisions  of  both  commanders  and 
warfighters  that  will  guide  their  choice  of  the  appropriate 
tactics  in  the  appropriate  context. 
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